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An ePortfolio Assessment Institute (AI) structured as a faculty development opportunity was 
undertaken to increase faculty confidence in teaching and assessing ePortfolios and to collect 
reliable data about student performance on four learning outcomes associated with an institution¬ 
wide ePortfolio initiative. Faculty raters participated in the two-day AI and received more than a day 
of training to use a summative rubric consistently. Faculty were asked to rate their own confidence 
in teaching and scoring each of the outcomes before coming to the AI and at the end of the AI. 
Generalizability-theory was used to estimate rater pair consistency. After establishing that the data 
were reliable, we analyzed the data to reveal a wide range in performance across ePortfolios. The 
survey of faculty showed statistically significant improvement in confidence across both teaching 
and evaluating for all outcomes. The study thus demonstrates that structuring an AI as a professional 
development activity increases faculty confidence in teaching and assessing outcomes related to 
ePortfolios. The study also demonstrates that ePortfolio initiatives can be successfully assessed even 
if commercial platforms that standardize and privilege assessment are not used and the ePortfolios 
themselves remain in the control of students rather than the institution. 


ePortfolios have recently been named by the American 
Association of Colleges and Universities (AAC&U) as a 
high-impact practice (HIP) because of the substantial 
evidence that they have an impact on student learning across 
a number of domains (Watson, Kuh, Rhodes, Light, & Chen 
2016). For example, Buzzetto-More (2010) demonstrated 
that ePortfolios allow students to synthesize their learning 
experiences, connect their course work to real world 
practices, consider what evidence demonstrates their skills 
and abilities, and compose reflective descriptions that build 
metacognition. As Watson et al. (2016) point out, however, 
“the keys to employing ePortfolios as a HIP are effective 
implementation and integration” (p. 67). Professional 
development activities that support faculty as they integrate 
ePortfolios into the curriculum are essential to both 
implementation and integration (Eynon & Gambino, 2016). 

Because the process of creating an ePortfolio can 
have an impact on students, our public, higher-research 
activity, land-grant university enrolling a total of 
28,000 students in undergraduate, professional, and 
graduate programs chose ePortfolios as the Quality 
Enhancement Plan (QEP) for our Southern Association 
of Colleges and Schools Commission On Colleges 
(SACSCOC) reaffirmation in 2013. Because we knew 
the importance of faculty involvement in implementing 
ePortfolio thinking throughout the curriculum, we 
designed our ePortfolio Project (hereafter Project) to 
include significant attention to faculty development and 
support (Bhika, Francis, & Miller, 2013; Hoekstra & 
Crocker, 2015) through a Faculty Cohort, essentially 
creating a Faculty Learning Community (FLC), a 
structure that has been shown to have a positive impact 
on student learning (Herman & Crowley, 2014; Jetton, 
Cancienne, & Greever, 2008; Smith et al., 2008) and on 


faculty (Cox, 2013; Cox & Richlin, 2004; Nadelson, 
2016; Wagner et al., 2015). 

From the beginning, our vision has been to provide 
students and faculty alike with a rich learning 
opportunity. For students, ePortfolios create an 
occasion to reflect on curricular and co-curricular 
experiences, discover common threads throughout those 
experiences, and articulate the meaning and 
significance of those experiences to themselves and 
professional audiences in a holistic way. Creating this 
kind of ePortfolio allows students to practice the higher 
order thinking of synthesis and evaluation (Peet et al., 
2011). For faculty, ePortfolio implementation invites 
consideration of what students from their program 
should be able to showcase, where the skills they expect 
students to demonstrate are taught, where students 
receive feedback that guides and redirects them, and 
where individual courses overlap and connect to 
contribute to the educational experience. Structured to 
encourage such reflection and collective conversation 
by faculty, an ePortfolio initiative creates an 
opportunity for faculty to reconsider programmatic 
priorities and values and then to reexamine the 
curriculum to discern the extent to which it aligns with 
and supports those priorities and values. For instance, if 
faculty members determine that they want graduates 
from their program to demonstrate an ability to 
communicate to different kinds of audiences, then 
faculty are forced to consider where in the curriculum 
they are teaching students to do such work and giving 
them opportunities to practice before expecting 
masterful performances. ePortfolios are both a process 
and a product, and we believe that the reflective work 
that happens throughout the process of both creating 
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ePortfolios and implementing ePortfolio thinking is 
valuable, whether or not external audiences look at the 
ePortfolio-as-product. 

Many universities that have begun ePortfolio initiatives 
use ePortfolios as an assessment tool (e.g., see the 
descriptions at http://c21.mcnrc.org/category/campus- 
stories/outcomes-assessment-stories/). Because their focus is 
on other learning outcomes—often those of general 
education or required for professional certification— 
ePortfolio initiatives that focus on assessment usually ask 
that students include similar documents, limit the design 
decisions that ePortfolios can invite, and may give less 
attention to the value-added experience of composing the 
ePortfolio or the curricular and pedagogical adjustments that 
ePortfolios can require of faculty. Such assessment-driven 
ePortfolios can be seen by students and faculty as 
bureaucratic requirements rather than opportunities for 
additional learning. Assessment-driven ePortfolios are 
typically “owned” by the institution so that they remain 
stable over time, and if students want to use their work for 
an external audience as they seek post-graduation 
employment or entry into advanced studies, they must often 
construct a separate ePortfolio that can remain in their own 
control. Though ePortfolios can have the additional benefit 
of exposing students to issues of professional identity, 
conventions expected by different professional audiences, 
and issues of visual, technical, and ethical literacy, they do 
so best when the ePortfolio is framed as a vehicle for 
representing themselves and their learning experiences to a 
professional but external audience. But professional identity, 
audience expectations, and the ethical literacy at play in 
crafting an integrated and professional representation are 
complicated issues which are more difficult to manage 
when assessment is privileged over individual choices 
because the difficult decisions students need to make are too 
often stripped away in the name of stability or consistency 
of assessment data. 

In choosing outward-facing, integrative, professional 
ePortfolios as our institution’s QEP, we opted for: 

• privileging student choice and ownership over 
ease of assessment; 

• using free platforms rather than expensive 

ones that claim backend assessment 

functionality but limit individual choices; 

• encouraging the creation of unique 

professional identities instead of requiring 
standard templates that promise to make 
evaluation more consistent; 

• inviting faculty in all disciplines to think 

through the messiness of teaching both visual 
literacy and the ethical considerations 

introduced when digital technology is made 
public whether or not they think of their 
discipline as visual or requiring advanced 

technical skills; and 


• asking faculty and students alike to 
reconsider what they think about effective 
communication and critical thinking when the 
audience is both public and professional but 
not necessarily academic. 

We admit that ePortfolios created as professional 
but personal websites controlled by individual students 
raise difficulties for institutions needing to assess these 
sites as evidence of student learning. In choosing to 
privilege student learning and student ownership, we 
also chose to grapple with those assessment challenges 
as additional opportunities for faculty development and 
engagement. We do not regret the choices we made, 
and in other contexts we have provided evidence that 
our Project has supported both effective implementation 
and integration into the curriculum (Bartlett, Stuart, 
Owensby, & Davis, 2016). We believed initially that 
faculty in the disciplines would be able to evaluate their 
students’ ePortfolios using their own deep 
understanding of disciplinary expectations and their 
familiarity with the careers their students pursued. We 
assumed that faculty would generate assessment data 
for our Project by including in their evaluations the 
learning outcomes we had identified as most connected 
to the choices students would make in creating an 
outward-facing, integrative, professional ePortfolio, 
namely: critical thinking through reflection, visual 
literacy, technical competency, and effective 
communication. But as our Project developed, we 
recognized that the assessment data we were able to 
collect from departments were problematic. This article 
describes how we reorganized our assessment of 
student ePortfolios through an ePortfolio Assessment 
Institute (AI) in order to generate both reliable data 
about student performance and as another opportunity 
for faculty development. We report on the evidence of 
our success in achieving both these goals despite the 
challenges of student control, individuality, and the lack 
of consistency caused by privileging learning over 
assessment. We detail here how we prepared for and 
organized the AI, created and then trained faculty to use 
a summative rubric, and produced data about student 
performance that are reliable. In short, we demonstrate 
that it is not only quite possible to assess ePortfolios 
and the learning objectives associated with ePortfolio 
projects without having a platform do it for you, but 
that the activity of assessment can further faculty 
engagement, thinking, and confidence in teaching and 
evaluating ePortfolios. 

Background 

In this section, we outline the institutional context 
for our Project, the work done to revise an initial rubric 
for assessing the four learning outcomes associated with 
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our Project, and the difficulties with our assessment data 
that led to the creation of the ePortfolio AI. 

Institutional Context 

Our Project was selected to be the university’s 
QEP in part because it built on a university-wide 
writing initiative that began in 2010. The writing 
initiative was the result of a faculty task force charged 
with investigating more than 10 years of National 
Survey of Student Engagement (NSSE) data in which 
students consistently reported having fewer writing 
assignments than their peers at comparable institutions. 
After two years of comprehensive work, the task force 
recommended creating an Office of University Writing 
(OUW) that would help faculty embed significant 
writing experiences in every undergraduate major and 
offer students support through an expanded writing 
center. Departments created writing plans to integrate 
writing into existing courses and submitted those plans 
for review and approval by a faculty committee. In 
reviewing these plans, it became clear that many 
programs were asking students to complete multimodal 
writing assignments, synthesize learning experiences in 
capstone-type courses, and in some cases, create a 
personal website or portfolio. Unfortunately, little 
infrastructure existed to support faculty and 
departments in these efforts. Our Project thus aimed to: 

• build on existing efforts; 

• expand the ways that the writing initiative had 
already begun to address the institutional 
concern that more attention to communication 
skills was necessary; and 

• provide additional support for faculty as they 
embedded ePortfolios and the reflective 
writing such personal websites require 
throughout the curriculum. 

We chose to focus our Project on integrative, 
outward-facing, professional ePortfolios that students 
would complete by the time of graduation, but because 
of the diversity of programs in our institution, we built 
in structures that would let programs use ePortfolios in 
a variety of ways. Some programs join our Project as 
they begin to think about whether ePortfolios would be 
useful to their students and their curricular objectives. 
Other programs join when they have already decided to 
require students to complete a senior ePortfolio. 
Programs also join with different numbers of faculty 
involved in integrating ePortfolio thinking into the 
curriculum, guiding students in producing ePortfolios, 
or assessing the results. Our Project is opt-in, and 
students who want to complete an ePortfolio but who 
are enrolled in a major that has not joined our Project, 
have the support of the OUW—which serves as the 


administrative home for the Project and offers programs 
for faculty and students—as well as the Career Center, 
the Writing Center, and the Media and Digital 
Resources Lab. 

Student Learning Outcomes (SLOs) 

In developing our Project, we drew on examples 
from other institutions and the research conducted by 
the Inter/National Coalition for Electronic Portfolio 
Research (ncepr.org). Because we were committed to 
using ePortfolios as an additional learning experience 
rather than as a tool to measure learning experiences 
that happened elsewhere, we identified outcomes that 
would happen as a result of creating an ePortfolio and 
settled on four student learning outcomes: (1) critical 
thinking through reflection, (2) visual literacy, (3) 
technical competence, and (4) effective communication. 
For our on-site SACSCOC visit we prepared an initial 
rubric for these outcomes modeled loosely on the 
AAC&U’s Valid Assessment of Learning in 
Undergraduate Education (VALUE) rubrics (see 
AAC&U, 2017). This initial rubric was treated as a 
beginning point and programs that joined our Project 
were actively encouraged to make it more specific to 
their needs or expand it to include additional items. As 
our Project grew and faculty learned more about the 
kind of thinking ePortfolios encourage, our outcomes 
began to have more specific elements. When faculty 
engaged in conversations about the examples students 
produced, we began to distinguish not only the different 
outcomes and the elements within those outcomes, but 
different levels of performance, as well. Effective 
communication, for example, initially seemed to mean 
everything and too often relied on the reader 
understanding the content of specific documents or 
artifacts students included rather than on effective 
communication across the entire ePortfolio. Over time, 
effective communication began to reference more 
regularly the consistency of choices for a chosen 
audience and the creation of a coherent story that 
provided evidence via the artifacts chosen to support 
claims made by the student author about their 
experiences and skills. Critical thinking through 
reflection likewise narrowed to refer not to every choice 
the student made, but only to the way in which 
reflective thinking was made visible in the contextual 
prose students wrote for the artifacts they included. 

Revising the Project Rubric 

As our Project grew, it became clear that programs in 
our faculty cohort were handling assessment in very 
different ways. We thus undertook a systematic process to 
observe program-level assessment of ePortfolios, including 
talking with faculty responsible for assessing ePortfolios and 
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reviewing program modifications to the initial ePortfolio 
rubric. We noted that assessment practices within a program 
became more sophisticated as student work improved, as 
more faculty became familiar with ePortfolio practices, and 
as these faculty members had more opportunities to develop 
a shared culture of expectations. We used what we learned 
about program-level assessment practices in two different 
ways: first, we created opportunities for faculty to share 
what they were doing with other members of the faculty 
cohort; and second, we began to revise the initial rubric to 
better reflect developing expectations and deeper levels of 
understanding. 

The initial rubric was also used by a faculty 
committee to recognize exemplary student work for an 
annual ePortfolio award, but we could see the 
possibilities and limitations of that initial rubric in this 
context, as well. On the one hand, the initial rubric 
created a framework for faculty from different 
disciplines to evaluate student work, discuss their 
expectations and judgments, and decide which students 
to recognize. On the other hand, the committee 
members needed training, practice, and discussion in 
order to use the rubric consistently. With changes in 
committee membership each year, a training process 
would help individuals understand the outcomes and 
apply the performance criteria to ePortfolios from very 
different disciplines, setting aside their own disciplinary 
expectations and content knowledge to concentrate on 
the performance across the ePortfolio. Like the faculty 
working to evaluate ePortfolios in programs, the awards 
committee was developing more refined expectations 
for student work, expectations that the original rubric 
did not always or consistently reflect. 

The revised rubric, now referred to as the formative 
rubric (see Appendix A), went through multiple 
iterations during the summer of 2015 and was repeatedly 
tested on existing student ePortfolios. The formative 
rubric was also circulated among members of the faculty 
cohort and committee members. We asked faculty to try 
the rubric on student ePortfolios they had access to or on 
the ones we made available in a gallery on our Project 
website. We used the feedback to reshape both the 
substance of the rubric and the way it was designed, 
crafting the formative rubric as both a teaching tool and 
an evaluative instrument. The newly revised and 
redesigned formative rubric was launched in the fall of 
2015, with faculty discussions and workshops focused on 
explaining the revisions, but not systematically training 
faculty to use it for consistent assessment. 

At the same time, we were rethinking the way in 
which we were collecting assessment data from 
programs. Because assessment was being done so 
differently across different programs, we were uncertain 
that the data provided to us by programs were reliable or 
consistent enough to guide decisions at the university- 
level or to serve as adequate evidence in our mid-cycle 


report to SACSCOC. In crafting the formative rubric as a 
teaching tool, we had included language about ePortfolio 
creation processes and eliminated specific behavioral 
anchors tied to features that could be observed directly in 
the ePortfolio. We worried that these choices would 
make the formative rubric harder to use for the purpose 
of consistent assessment. 

In preparing for the AI, we conducted a test-day 
with 10 faculty and four student ePortfolios. Our goal 
for the test-day was both to refine the training process 
we would use at the AI and to test the effectiveness of 
the formative rubric when used by multiple raters. Sure 
enough, scores from the test-day did not achieve inter¬ 
rater reliability. Based on faculty feedback, we 
concluded that the lack of behavioral anchors in the 
rubric and the nine levels of performance included in 
the formative rubric were contributing to the lack of 
consistency in scores. Recognizing that a different 
rubric was needed to produce reliable assessment data, 
we undertook another rubric revision to resolve these 
problems and created what we now refer to as the 
summative rubric (see Appendix B). 

Based on the outcomes and descriptors outlined in 
the formative rubric, we created an initial draft of the 
summative rubric, with only four levels of performance 
and observable behavioral anchors for each clement. In 
multiple sessions during early spring of 2015, a team of 
four to five individuals from the OUW collaborated to 
draft and test iterations of the summative rubric using 
existing student ePortfolios. Before each session, 
members of the team would individually rate student 
ePortfolios and highlight sections of the rubric that 
needed more work. Each time the team met, scores 
were shared and the rubric was discussed and 
collectively revised. Conversations during these 
sessions included differing interpretations of meaning at 
specific points in the rubric, missing or incorrect 
language in descriptors, and individual perceptions of 
student work and how these impacted the evaluation 
scores. These discussions were also considered in 
relation to the test-day training session and the training 
planned for the AI; we knew we would need examples 
that would elicit rich discussion of the rubric and allow 
faculty to practice recognizing specific elements of the 
outcomes in various ePortfolios. Once the revisions of 
the summative rubric were complete, the team tested it 
on multiple ePortfolios until intcr-rater reliability was 
achieved across ePortfolios from a variety of 
disciplines. Finally, the summative rubric and the 
planned training process was tested in April of 2016 
with the Awards Committee. 

Collecting Reliable Data 

In addition to the problems created by a rubric that 
was less-than-ideal for consistency in assessment, our 



Marshall, Bartlett, Duffy, and Powell 


Increasing Faculty Confidence 191 


processes for collecting the data that resulted from 
program-level evaluations of student ePortfolios were 
fraught with difficulties. For example: 

• Though some programs had faculty 
collaborate to assess student work, faculty in 
other programs did not always agree on how 
(or whether) to use the rubric; 

• Not all programs conducted training in using 
the rubric, and even where they did, not all 
faculty participated in these norming exercises 
to ensure that the rubric was being used 
consistently across different faculty raters; 

• Programs did not submit the evaluation data in 
the same format; sometimes grades were 
submitted rather than rubric-guided scores; 

• Faculty in administrative positions often assumed 
responsibility for assembling the data from their 
program and reporting it to the OUW, but 
changes in these department-level leadership 
positions meant that there was confusion about 
what data were needed and how these requests 
for data were different from other institutionally- 
required assessment reports; 

• As our Project grew, the numbers of programs 
that needed to be asked for data on a regular 
basis also grew, but with a predictable range of 
positive and negative responses. 

Taken together, these logistical problems created gaps 
in our data and a growing reluctance to trust the data as 
reliable indicators of student performance. We 
determined that an AI would be an alternative way of 
collecting direct evidence of student performance on 
our four outcomes that could resolve these difficulties. 
We enlisted the expertise of the Director of Academic 
Program Assessment to help us design the AI, modeled 
after similar AIs used for other purposes. 

Method 

Our AI was designed as a two-day faculty 
development event held after graduation in May 2016. 
Because we planned to publish the results, we sought 
and received Institutional Review Board (IRB) approval 
to recruit faculty whose programs were already 
participating in our Faculty Cohort and students who 
we believed would have created an ePortfolio since the 
beginning of our Project. An email invitation with a 
consent document was sent to selected Faculty Cohort, 
members chosen to reflect the diversity of disciplines 
participating in our Project. The faculty members who 
received the original email were asked to nominate 
another faculty member from their department whom 
they would like to have as their partner, preferably 
someone who was not already deeply involved in our 


Project. These new nominees then received an 
invitation to join the AI in which the other person from 
their department who had agreed to participate was 
named. We thought this approach would lessen 
potential personality clashes because faculty would 
have a voice in identifying a colleague with whom they 
would feel comfortable and whom they felt was likely 
to be interested in knowing more about our Project. We 
aimed to deepen participation across the programs by 
including faculty who had not already been active in the 
Faculty Cohort and we wanted to see if the level of 
engagement of the faculty and their familiarity with our 
Project influenced their scores. However, we saw this 
first AI as merely laying the ground work for potential 
studies more carefully focused on faculty and their level 
of engagement with professional development activities 
connected to our Project. 

Faculty participants were compensated $1,000 for 
completing both days of the AI. A total of 34 
participants served as raters for the AI, most from 
programs already in the Faculty Cohort. To fill in last 
minute withdrawals and strengthen collaborations with 
other units responsible for faculty development and 
assessment, we included a total of five professional 
staff members from the OUW, including two who were 
responsible for leading the training, two from the 
Center for Teaching and Learning, and one from the 
Office of Academic Assessment. 

To solicit student ePortfolios, we compiled a 
contact list of university students who met one or more 
of the following conditions: 

• graduated between August 2012-May 2016 
from a program that had joined the faculty 
cohort; 

• attended workshops related to our Project; 

• received a nomination for an ePortfolio award; 

• held a leadership position as an ePortfolio 
Ambassador; and 

• served as a writing center tutor trained to help 
others with ePortfolios. 

A total of 705 students were contacted through 
their student email addresses and invited to participate. 
The email (Appendix C) included a link to a survey that 
served as the electronic consent. The survey (Appendix 
D) asked students to select which existing demographic 
data—such as major, grade point average (GPA), 
transfer and first generation status, scores from the 
American College Testing or Scholastic Assessment 
Test (ACT/SAT), ethnicity, gender, etc.—they were 
willing to have us access. The survey also asked 
students how they had used their ePortfolios, and when 
they had completed it, and allowed them to provide one 
or more Uniform Resource Locators (URL) if they were 
willing to allow us to use their ePortfolio in the AI. To 
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encourage participation in the study, student 
participants were eligible to pick up a promotional item 
valued at less than $5 from the OUW. In addition, each 
student who completed the survey link in the email was 
entered into a random drawing to receive a $50 
Amazon gift card. A total of 79 students responded and 
completed the survey (11.2% response rate), with 61 
students providing a URL to their ePortfolio and 
consenting to have it used at the AI. We identified 
several factors that could have led more students to 
answer the survey than were willing to provide us with 
a URL. First, students would have needed to maintain 
their ePortfolio after graduation, or at least remembered 
the URL they had used. Second, some students would 
have started ePortfolios in courses, but not all would 
necessarily have finished them. Finally, even though 
students had nothing at risk in how their ePortfolios 
were evaluated, we suspect that students need a certain 
level of pride in and confidence that their ePortfolios 
were good in order to give faculty access to them. 

Before arriving at the AI, faculty raters were asked 
to complete a survey (see Appendix E) indicating their 
confidence in teaching and evaluating each of the four 
learning outcomes associated with our Project. The 
survey was designed on a 5-point Likert scale (1 = not 
confident at all, 5 = highly confident). At the end of the 
AI, raters were asked to complete the survey again, 
answering these same questions and providing feedback 
about the training and the overall experience. Faculty 
raters had not seen the summative rubric until they 
arrived at the AI, though those who had been 
participating in the faculty cohort had seen the 
formative rubric. 

Because we had some faculty raters who were new 
to our Project and to ePortfolios, the training process 
included an overview of the learning outcomes 
associated with our Project. The purpose of the training 
exercises was to elicit discussion of features we had 
learned would create difficulties for at least some raters 
and to develop a common understanding of the 
elements within each outcome and the differences in 
performance represented by the rubric. The schedule for 
the AI is included as Appendix F. Other training 
components on the first day included: 

• writing and discussion of the assumptions and 
experiences individuals have of ePortfolios to 
move faculty raters to set disciplinary 
expectations or personal preferences aside and 
rely on the language of the rubric; 

• individual and small group work on key terms in 
the rubric to ensure that everyone was familiar 
with the rubric and had thought about the language 
used across the different levels of performance; 

• guided practice in using the rubric with a 
single ePortfolio with time to read through the 


ePortfolio before scoring one outcome at a 
time. Key points for each outcome were 
identified and participants were encouraged to 
share their interpretations and the rationale for 
the scores they gave; 

• individual practice with an ePortfolio with 
raters having 30 minutes to read and score and 
then 50 minutes for the whole group to 
compare scores and discuss so that elements of 
the outcomes and performance levels became 
clearer and typical issues of difference were 
considered; and 

• a wrap-up exercise that asked participants to 
consider how they might use anything that had 
happened in their first day in their own 
programs or courses. 

The second day of training began with a brief 
discussion to answer any questions participants had and 
a norming session where a single ePortfolio was scored 
by all raters. This ePortfolio had also been scored in 
advance by the AI organizers and so was used as a 
control that would serve as an anchor score for 
comparative purposes in analysis. In this norming 
session, raters were reminded that they were allowed to 
give half points on the 4-point scale. Once the scoring 
was completed, all scores across all elements of the 
rubric were within one point on this norming sample. 

Throughout the training process, we emphasized 
returning to the language of the rubric, grounding 
judgments in the rubric rather than in individual 
preferences or disciplinary expectations. For example, 
we were especially careful to talk about the difference 
in a student’s statements of religious faith and the 
consistency and judgment in making such statements 
for the intended audience. The emphasis on 
interpretation and reading practices was confusing at 
first to some faculty, especially those from science, 
technology, engineering, or mathematics (STEM) 
disciplines that rely on quantitative data and who 
assumed that scoring with a rubric would not involve 
interpretations of either the rubric language or the 
students’ choices. 

Because we are interested in the learning that 
happens as students create an ePortfolio, we encouraged 
raters to see the ePortfolio holistically and to check on 
artifacts included in the ePortfolio only to the degree 
that they provided evidence to support claims or 
matched reflective contextualizing. We also prioritized 
reading the ePortfolio holistically because our 
experience suggested that few readers—employers or 
faculty—spend time with each artifact. We regularly 
asked faculty to set aside their disciplinary practices or 
assumptions to consider ePortfolios that were outside 
their discipline as capable of demonstrating the 
elements on the rubric. This was especially difficult in 
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the case of visual literacy, since we have many 
programs that are related to the visual arts or design, 
including studio arts, architecture, graphic and 
industrial design, apparel design, and interior design. 
Raters in these design-related disciplines are 
accustomed to looking for particular visual features and 
have strong opinions about what constitutes “good” 
design. On the other hand, raters who are not in design- 
related disciplines too often assume that students in 
their disciplines have nothing visual to include and little 
need to attend to design choices. Such faculty raters 
either ignore design choices or are overly impressed 
with any visual content. 

Finally, we asked raters to see the topmost 
performance level as aspirational and use it only for 
truly exceptional performances. During the training we 
modeled negotiating different scores, which would be a 
part of the adjudication process at the end of the 
individual scoring, by having raters who scored above 
or below the majority of participants explain their 
thinking. We let raters change their scores both during 
the training and during the adjudication process, but we 
did not require them to do so. During the training 
process, however, all raters could see the collected 
scores during the discussion and the entire group could 
see where we were scoring consistently and where we 
were not. During the adjudication process, however, 
teams could see only their own scores and did not have 
any information about how their scores matched or 
differed from those of other teams. 

Following the rubric training, each rater was 
provided with eight scoring sheets (Appendix G) that 
matched the summative rubric. Each scoring sheet 
contained a unique URL for the ePortfolios the rater 
would be assessing during the AI. Laptop computers 
were provided for raters to access the ePortfolios. 
Raters were asked to work independently to score eight 
ePortfolios. As noted above, 61 students provided a 
URL to their ePortfolio, consenting to have it scored at 
the AI. Each of these ePortfolios was scored by at least 
two teams of raters at the AI, creating at least four 
scores for each ePortfolio. When distributing the 
ePortfolios, we aimed to lessen the interference of 
disciplinary knowledge and to eliminate bias from 
knowledge that raters might have of students outside 
the ePortfolio. Therefore, we assigned ePortfolios to 
raters who were not in the same department or a closely 
aligned discipline. Before the scoring began, we also 
asked raters to identify any student they knew 
personally from whatever context and then reassigned 
those ePortfolios to a different rater. Raters were 
reminded that ePortfolios could have been created at 
different points in the student’s career and under 
different conditions. Therefore, the raters were to 
evaluate the evidence of the four learning outcomes 
associated with our Project rather than the potential of 


the student or the limits imposed by the conditions of 
creation. Raters were encouraged to take notes during 
the evaluation of each ePortfolio that might be useful to 
them during the adjudication with their team member. 
As raters completed their scoring of each individual 
ePortfolio, they were asked to submit the scoring sheet. 
We believed immediate submission would lessen the 
temptation to score by comparing ePortfolios rather 
than by relying on the rubric. A lunch break was 
provided, but raters were asked not to talk about any of 
the ePortfolios they had scored in order to lessen the 
chance that comments would influence raters who were 
yet to score that ePortfolio. 

At the end of the scoring session, scoring sheets 
were returned to the teams, and they were asked to 
adjudicate any differences, but to concentrate on 
elements where their scores were more than one point 
apart. The adjudication process afforded each pair of 
raters the chance to discuss why each ePortfolio 
received the score it was given, and to connect those 
scores to specific language in the rubric. The primary 
goal of the adjudication process was to determine why 
the ePortfolio received different scores from each rater, 
and then move toward consensus about what the rubric 
means. It was not our explicit intention to have raters 
change their scores, but rather to determine whhether 
the raters missed something, deviated from the rubric, 
or became convinced that their interpretation the rubric 
or the ePortfolio was incorrect. 

A debrief discussion at the end of the second day 
served as the wrap-up and was framed as helping the 
organizers to plan the next AI and create follow-up 
programs for Faculty Cohort members. Following our 
IRB approved protocol, any notes faculty had made, all 
scoring materials, and the list of ePortfolios assigned to 
each rater were collected, and URLs were erased from 
the laptop computers raters had used. The statistical 
analysis on the data from the surveys completed by the 
raters, as well as the student ePortfolio scores supplied 
by the raters, was conducted through the use of 
Statistical Package for the Social Sciences (SPSS). 
Statistical Analysis System (SAS) was used to conduct 
G-score analysis on the reliability of the paired raters. 

Results 

Reliability 

We began our analysis by exploring whether the 
data provided by rater teams were reliable. In other 
words, were the scores given by raters a reflection of 
consistent use of the summative rubric? 
Generalizability theory was used to determine G- 
coefficients for each rater team. These coefficients are 
reliability estimates, ranging from 0-1, with higher 
estimates reflecting greater reliability. Typically, 0.70 is 
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Figure 1 

G-Coefficients for Paired Teams Based on all ePortfolios Scored by the Team 



3 15 10 16 12 8 14 11 2 1 6 13 7 5 4 9 17 


Table 1 


Comparison of Means Using all Scores To Only Scores From Most Reliable Teams 


Rubric element 

All data 

Most reliable teams 

Critical thinking through reflection 

A. Artifacts 

2.52 

2.55 

B. Arrangement 

2.37 

2.43 

C. Reflective writing 

2.21 

2.31 

Visual literacy 

D. Visual elements 

2.38 

2.34 

E. Design choices 

2.33 

2.38 

Technical competency 

F. Navigation 

2.62 

2.66 

G. Attention to technical details 

2.63 

2.61 

H. Ethical literacy 

2.13 

2.16 

Effective communication 

I. Coherent message for intended audience 

2.48 

2.51 

Overall average 

2.41 

2.44 


Note. The scale is 1 = beginner, 2 = developing, 3 = mature, and 4 = professional. 


a general cut-off for acceptable reliability. Figure 1 
shows the G-coefficients for each paired team, based on 
all of the ePortfolios they rated. Eleven of the 17 teams 
had a G-coefficient greater than 0.70, three teams were 
very close to 0.70 (i.e., 0.68), and three teams had less 
consistent scoring patterns. 

Given that some of our teams were more reliable than 
others and that all ePortfolios were scored by more than 
one team, we needed to make a decision about whether to 
base future analysis on the average score of all teams that 
scored a particular ePortfolio or to use only the scores 
from the more reliable team. To determine whether 
shifting procedures would make a difference in the scores, 
we did a comparison of the different means across each 
element on the rubric. Since some ePortfolios were scored 
by the same two teams and others were scored by 


randomized teams, we separated the data for each 
ePortfolio and chose only the scores provided by the teams 
with the highest G-score. The scores from the most 
reliable teams were then averaged to determine the mean 
for each element. Table 1 shows the mean scores for each 
element, first by using the scores of all raters who scored 
each ePortfolio and then by looking at only the scores 
from the most reliable team who scored each ePortfolio. 
Our analysis shows the means from the most reliable 
teams for most elements is only slightly higher than the 
means from all of the raters. 

Faculty Confidence 

We turn now to the question of whether the AI itself 
had an impact on the faculty who participated in terms of 
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their confidence in evaluating or teaching each of the 
outcomes associated with our Project. Our pre and post 
surveys asked faculty simply to indicate their level of 
confidence in teaching and assessing the four student 
learning outcomes using a 5-point Likert-type scale. 
Figure 2 below shows the means for each question. 
Because the assumption of normality was violated and 
the data were not distributed in a typical bell-curve, the 
use of a t test would be inappropriate. Instead, a 
Wilcoxon signed-rank test (the nonparametric equivalent 
of a t test) was conducted. The Wilcoxon signed-rank test 
showed that the results of the post-survey were 
significantly different from the pre-survey (Z = -2.533, p 
= 0.010) and indicate a significantly higher level of 
faculty confidence in teaching and assessing the four 
student learning outcomes. The mean scores on the 
pre/post survey revealed an average increase of 0.69 
across all questions. In addition, we noted that every 
participant except one, a member of the least reliable 
team, had a higher level of confidence after the AI than 
before it began on every outcome and for both teaching 
and assessing. That one participant scored their 


confidence level prior to the AI as a 5 for every item, 
leaving no room for improvement on our scale. 

In addition, we invited faculty to provide other 
feedback in an open-ended question. Comments 
indicated that participants felt the AI was helpful to 
them as they considered teaching elements of 
ePortfolios in their own courses and evaluating the 
ePortfolios created by students in their programs. We 
found no significant difference in the increased 
confidence of faculty who had been participating in the 
faculty cohort for some time and those who were new 
to our Project. We therefore concluded that the 
increased confidence is likely attributed to the AI itself 
and that the AI functioned as an effective professional 
development opportunity for those who participated 
regardless of their prior experience with ePortfolios. 

Discussion and Conclusion 

Though the structure of our Project has focused on 
student learning as students create integrative, outward¬ 
facing, professional ePortfolios to represent themselves 


Figure 2 

Pre-Post Confidence Comparisons 


Survey Means 


4.6 


4.44 



I feel confident I feel confident I feel confident I feel confident I feel confident I feel confident I feel confident I feel confident 
teaching Critical assessing teaching Visual assessing Visual teaching assessing teaching assessing 
Thinking Critical Thinking Literacy in Literacy in Technical Technical Effective Effective 

through through ePortfolios. ePortfolios. Competency in Competency in Communication Communication 

Reflection in Reflection in ePortfolios. ePortfolios. in ePortfolios. in ePortfolios. 
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and their learning to an external audience, our decision 
to allow students to use a variety of platforms and 
maintain control of their own ePortfolios has created 
some challenging assessment issues. Our learning 
outcomes have remained consistent, but our ability to 
articulate what is involved in each of those outcomes 
and how those outcomes are made visible in ePortfolios 
has evolved and deepened as our Project has matured. 
We have seen faculty and students come to understand 
the possibilities of ePortfolios in more sophisticated 
ways as they create ePortfolios or integrate ePortfolio 
thinking into courses. In essence, both faculty and 
students change their understanding of, and their 
expectations for, the four learning outcomes by doing 
ePortfolios. Though we had an original rubric that 
included behavioral anchors, we would not have been 
able to create the formative or summative rubric earlier; 
we simply did not know enough until we all had gained 
more experience. Creating a formative rubric that 
serves as a teaching tool but that is less effective for 
collecting consistent assessment data was an important 
step in developing the kind of language necessary for a 
summative rubric and gave us specific examples of 
where readers can have difficulties evaluating student 
performance in ePortfolios. These examples were 
essential to the training we provided for faculty raters 
during the AI. Creating a summative rubric with four 
levels of performance and clear behavioral anchors for 
each element and each level of performance was 
necessary to generate reliable assessment data. Inviting 
faculty to participate in an ePortfolio AI and training 
them to use the summative rubric to score a variety of 
ePortfolios outside their own disciplines accomplished 
two important goals for our Project: (1) we were able to 
gather reliable assessment scores for each of the 
learning outcomes; and (2) we increased faculty 
participants’ confidence in teaching and evaluating 
those outcomes. 

We conclude from our analysis of G-coefficients 
that most of the raters were able to use the rubric 
consistently to score the ePortfolios they were assigned. 
Some teams were harsher or more lenient than others, 
but all but one of the teams achieved a reliability 
estimate near or above 0.70. We eliminated the one 
team (Team 3) that did not achieve reliability from all 
analysis. When comparing whether to use an average of 
all remaining teams who scored a particular ePortfolio 
or to average only the scores from the most reliable 
team that scored each ePortfolio, we determined that it 
would be better to use only the scores from the most 
reliable team, essentially eliminating the scores from 
the other two less reliable team members. We plan to 
continue the analysis of our data to better understand 
the relationship between such factors as GPA, test 
scores, involvement in our Project and scores assigned 


by raters at the AI. As we continue to analyze our data 
and consider the scores in relation to the survey 
answers students provided about their use of their 
ePortfolios and the demographic data they consented 
for us to access and compare, we will use only the 
scores from the most reliable team in each case. 

Our analysis of sources of error suggests that our 
training was effective enough to produce reliable data 
and that we can trust the scores that remain to be 
reasonable indicators of student performance. When we 
repeat the AI with new student ePortfolios, we will be 
able to compare those scores, assuming those scores also 
prove to be reliable, to see if students are improving 
across the four learning outcomes associated with the our 
Project. We will repeat the AI in May 2018 with student 
ePortfolios created between May 1, 2016 and May 1, 
2018. We then plan to do a comparative analysis to 
determine the extent of improvement in the quality of 
student ePortfolios over time. We recognize, however, 
that students who created ePortfolios at the beginning of 
our Project were likely to have been highly motivated 
and that as more students produce ePortfolios, we may 
see a larger range of performances. Likewise, those 
students who maintain their ePortfolios after graduation 
and the initial job search may be more likely to grant us 
access than those who abandon their ePortfolios, further 
skewing the range of performances away from a normal 
distribution. We are interested in conducting follow-up 
interviews with students who participated in this study to 
see if we can uncover the factors that motivate students 
to create and maintain an ePortfolio or to grant us 
permission to use it for assessment of our Project. 
Finally, we are interested in related studies focused on 
faculty, including how they use the experience of 
participating in the AI in their own teaching. 

Assessment is not the focus of our Project, but we 
believe we have established that reliable assessment 
data can be generated without asking students to utilize 
a standard platform or follow a rigid set of requirements 
about what artifacts to include as they integrate their 
experiences and present themselves to an external 
audience. The question of whether such institutional 
initiatives have any impact on faculty is also at least 
partially addressed in the data we have collected here, 
demonstrating that faculty confidence can be increased 
by structuring assessment activities as opportunities for 
faculty learning. 
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Appendix A 
Formative Rubric 


SPORIFOIIO PROJECT Auburn ePortfolio Rubric 

LEARN IT. LIVE IT. SHARE IT. 


EFFECTIVE COMMUNICATION 


The ePortfolio demonstrates strong and consistent communication skills 


Message: 

• The central message of the ePortfolio is consistently clear 

• Artifacts have been carefully selected to support a central message 
and consistently function as evidence that supports the claims made 
in the ePortfolio 

Audience and purpose: 

• The number and kinds of artifacts are well selected and demonstrate 
careful attention to audience and disciplinary expectations 

Revision and editing: 

All components of the ePortfolio show consistent attention to conventions 

and proofreading: 

• Artifacts have been revised to demonstrate mastery 

• Where unrevised artifacts are included, they have been purposefully 
selected to exhibit growth and their presence is explained 


These skills are rarely 
present and when 
attempted they are of 
poor quality. 

NOVICE 


The presence of these 
skills is variable and when 
present the quality of 
these skills is inconsistent. 

DEVELOPING 


These skills are 
consistently present and 
demonstrate mastery 
through high quality work. 

PROFESSIONAL 



Score or level for this student outcome: 


TECHNICAL COMPETENCY 


The author uses technical features to enrich the delivery of the message 


Navigation and user experience: 

Navigation is thoughtful and intuitive, adds to user experience, and 
demonstrates the ability to think about the user's needs: 

• Navigation guides the user within the ePortfolio, connecting artifacts 
and experiences across pages or relevant details 

• Components of the ePortfolio are clearly labeled and easily used 

• Information like resumes and contact information is easily located 

Intentional use of technology: 

Technical features of the site: 

• Effectively reinforce the central message 

• Convey a consistent professional identity to the intended audience 

• Demonstrate a thoughtful application of technical features (slide 
shows, scroll bars, hyperlinks, animation, etc.) 

Attention to technical details: 

Attention has been given to ensure that technical features work as intended 
and support the professional identity. Possible examples include: 

• Photos and graphics are of high quality with attention paid to size, 
resolution, and color 

• Artifacts that utilize video or sound are well edited and of good quality 

• Care has been taken to ensure the entire site works in different 
browsers and devices 

• Links are active and well labeled and artifacts are easily accessed 

• Strategies for making the ePortfolio accessible to different users have 
been utilized effectively 

Ethical literacy: 

There is clear consideration of intellectual property and fair use: 

■ When another's work is used, credit is given with correct formatting 

• There is reference to personal authorship and ownership of materials 

• There is consistent and appropriate use of others' likeness, work, and/ 
or images 

The content of the ePortfolio itself reflects an awareness of the public 
nature of the internet: 

• Careful consideration of privacy issues is evident 

• Anonymous individuals are not treated as objects in service of the 
author's message 


These skills are rarely 
present and when 
attempted they are of 
poor quality. 

NOVICE 


The presence of these 
skills is variable and when 
present the quality of 
these skills is inconsistent. 

DEVELOPING 


These skills are 
consistently present and 
demonstrate mastery 
through high quality work. 

PROFESSIONAL 


Score or level for this student outcome: 
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VISUAL LITERACY 


The ePortfolio uses visual elements to enhance the effectiveness of the site 


Message in visual elements: 

The author has utilized visual features to further develop elements of the 

ePortfolio and enhance the overall impact. Possible examples include: 

To enhance text descriptions of experiences 
To convey a message not immediately evident 
As a new way to represent meaning or insight 

To integrate experiences, explanations, and demonstrate connections 
To connect an overall message to individual artifacts 

Intentional use of visual elements: 

The author demonstrates the ability to evaluate effectiveness both within 

visual elements and across the site itself, demonstrating: 

• Consideration of how the audience will interpret the elements 
themselves 

• Consistent formatting from page to page 

• Layout uses white space, alignment, and placement to appropriately 
organize content 

■ Headings, subheadings, and paragraphs contribute to easy 
identification of elements and readability 

• Background and text color are aesthetically pleasing, consistent across 
the site, and contribute to easy scanning 

• Images and icons explain the relationships between these visual 
elements and the artifacts they accompany 

Difference from social media: 

The visual elements demonstrate: 

■ An overall professional identity reflecting careful consideration of 
purpose and audience that differentiates the ePortfolio from a social 
media site 

• A consistent understanding of how visual elements contribute to the 
audience's interpretation of the author's professional identity 


These skills are rarely 
present and when 
attempted they are of 
poor quality. 

NOVICE 


The presence of these 
skills is variable and when 
present the quality of 
these skills is inconsistent. 

DEVELOPING 


These skills are 
consistently present and 
demonstrate mastery 
through high quality work. 

PROFESSIONAL 


Score or level for this student outcome: 


CRITICAL THINKING THROUGH REFLECTION 


The ePortfolio demonstrates critical thinking through reflection across a 
variety of elements 


Selection of artifacts: 

The selected artifacts in the ePortfolio demonstrate the result of an 

evaluative process because they: 

• Represent various experiences both in and out of class in a coherent 
way 

■ Analyze the selected artifacts to demonstrate an intended meaning 

• Demonstrate the ability to examine how the artifacts contribute to the 
overall message of the ePortfolio 

• Successfully demonstrate the ability to curate a collection of artifacts 
that creates a clear sense of identity, purpose, and audience 

Arrangement: 

The organization of the ePortfolio: 

• Demonstrates careful arrangement of artifacts to draw meaningful 
connections across experiences 

• Represents an analytic process to determine placement of experiences 
based on significance and the order is consistent and effective 

Reflective writing: 

The writing throughout the ePortfolio: 

• Reinforces the central message of the ePortfolio 

• Justifies the presence of artifacts by explaining why they are included 

• Explains the relationship between experiences and their visual 
representations 

• Critically examines experiences, perceptions, interpretations, and 
identity 

• Demonstrates a professional identity that simultaneously integrates 
past experiences and projects into the future 


These skills are rarely 
present and when 
attempted they are of 
poor quality. 

NOVICE 


The presence of these 
skills is variable and when 
present the quality of 
these skills is inconsistent. 

DEVELOPING 


These skills are 
consistently present and 
demonstrate mastery 
through high quality work. 

PROFESSIONAL 


Score or level for this student outcome: 
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Appendix B 
Summative Rubric 


gPOMFOIIO project Summative ePortfolio Rubric 

LEARN IT. LIVE IT. SHARE IT. 


ePortfolios can take many forms, but for the purpose of assessing the outcomes associated with this project, we expect an ePortfolio to tell a coherent story about the student's 
learning experiences both in and out of classes, synthesize and present those experiences for a general, external, professional audience. ePortfolios of this kind provide evidence of 
skills and interests through a curated selection of artifacts and craft in the process a professional identity. 


Critical Thinking Through Reflection 

Critical Thinking Through Reflection focuses on evidence of critical thinking (analysis, synthesis, evaluation, creation) as it exists in within artifacts, arrangement, and reflective 
writing and across the ePortfolio as a whole. 


Beginner: 1 

Developing: 2 

Mature: 3 

Professional: 4 

A: Artifacts 

Included artifacts show little connection to 
the overarching story or the story itself is 
missing. Artifacts are not contextualized so 
their meaning is supplied more by the 
viewer than the author. There is little 
variety of skills, experiences, and learning 
represented and not enough evidence to 
support the claims being made. Most 
artifacts are of the same kind or from the 
same kind of experience (for example 
course papers or images of design work). 

Some artifacts contribute to the story being 
told, but some may not. The story is 
present, but limited and individual artifacts 
have little contextual information to 
support their inclusion. There is some 
variety of skills, experiences, and learning 
represented. While the overarching story is 
not supported by all of the artifacts, there 
are some moments where artifacts do 
substantiate the claims. 

Most artifacts provide evidence of the 
story being told and most support the 
claims being made. Artifacts are 
contextualized so that the reason for their 
inclusion is almost always clear. The 
artifacts provided demonstrate a variety of 
skills, experiences, and learning across a 
range of courses or co-curricular 
experiences. 

Artifacts provide strong evidence of the story being 
told and claims being made. Artifacts are well 
contextualized so that their presence in support of a 
message is clear throughout. The artifacts provided 
demonstrate a variety of skills, experiences, and 
learning and draw from a wide range of experiences 
both in and out of formal courses. 

B: Arrangement 

Arrangement is overly simplistic, for 
example organized by the course or level 
or presented as a simple gallery or list. 

There is no evidence of synthesis of 
learning and the arrangement of artifacts 
makes the overarching story confusing. 

Arrangement is mostly logical though 
predictable. Some items may be misplaced, 
disconnected, or underdeveloped. There is 
limited evidence of synthesis in learning so 
that the overarching story is vague or in 
places confusing. The reader has to do too 
much of the work to interpret the 
connections. 

Arrangement usually reinforces the story 
and the ePortfolio almost always functions 
as a curated collection of evidence. There 
is some evidence of synthesis of learning 
experiences through the use of repeated 
themes, links within the ePortfolio, or 
other motifs that are not overly cliched. 

Arrangement consistently reinforces the story. The 
ePortfolio functions as a carefully curated collection 
of evidence intentionally assembled to demonstrate 
synthesis of, and sophisticated thinking about, 
various learning experiences. The story is clear and 
nuanced. 

C: Reflective Writing 

Reflective writing is limited to description. 
Where attempts to go beyond description 
occur, they seem simplistic, formulaic, or 
cliched. The writing is often repetitious and 
may have numerous errors at the sentence 
level. 

Reflective writing is present and 
occasionally includes more than description 
by connecting artifacts together, saying 
why an experience was important, or 
connecting lived experience to other 
sources of knowledge. The writing is almost 
always correct, but does not fully construct 
a coherent individual identity. 

Reflections often make connections, 
explains importance, or project into the 
future, but not consistently so. Where the 
writing indicates critical thinking, it does so 
by making connections to specific artifacts 
and to the story being told across the 
ePortfolio as a whole. The writing is 
generally correct and constructs a sense of 
individual identity, though at times it is 
overdone or re pit it io us. 

Reflections are insightful and work together to 
consistently synthesize learning experiences and 
demonstrate critical thinking about the meaning and 
application of these experiences without being 
overdone or repetitious. Across the whole ePortfolio 
the writing constructs a sophisticated and nuanced 
identity. 


AUBURN UNIVERSITY 


Office of University Writing 

www.auburn.edu/writing 

auburnwrites@auburn.edu 

5/4/16 


OFFICE OF UNIVERSITY WRITING 














Marshall, Bartlett, Duffy, and Powell 


Increasing Faculty Confidence 


202 


GPOMFOIIO project Summative ePortfolio Rubric 

LEARN IT. LIVE IT. SHARE IT. 


Visual Literacy 

Visual Literacy focuses on how the author uses visual elements to provide evidence, construct deeper meaning, and support and enhance the message of the ePortfolio. "Visual 
elements" refer to any non-text elements such as boxes, icons, buttons, or photographs. 


Beginner: 1 

Developing: 2 

Mature: 3 

Professional: 4 

D: Visual Elements 

Visual elements decorate the ePortfolio 
without attention to how they function. 
They do not support the message or help 
create identity. 

Visual elements illustrate the message but 
inconsistently or visual elements exist mostly 
because they are artifacts of design or artistic 
creation. In either case, visual elements do not 
consistently contribute to the message or 
identity. 

Visual elements function to illustrate the 
message or provide evidence of experiences or 
skills, and almost always sync together with the 
text to provide additional meaning and craft an 
identity maintained throughout the ePortfolio. 

Visual elements are used both to illustrate and 
provide evidence, but always with attention to 
syncing the image and the text and to 
constructing an appropriate identity. The visual 
elements included are unique, creative, 
sophisticated, and convey another layer of the 
meaning without functioning simply as artifacts. 

E: Design 
choices 

Design choices like color, font, space, and 
layout are haphazard and distract from 
the message and identity. 

Design choices are often appropriate, but 
some choices of color, font, space, or layout 
are ineffective in supporting the message or 
distract from the identity being created. 

Design choices are consistently appropriate, 
support the message, and aid in the 
construction of identity. Some of these choices 
are sophisticated, original, or creative. 

Design choices are consistently appropriate, 
support the message, and aid in the creation of 
identity. These choices are consistently 
sophisticated and original/creative. 
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BPOMFOUO project Summative ePortfolio Rubric 

LEARN IT. LIVE IT. SHARE IT. 


Technical Competency 

Technical Competency focuses on the application of technical elements that should enhance the way information is conveyed to an audience, differentiating an ePortfolio from 
other products (social media sites, blogs, commercial websites) to construct identity. "Navigation" refers to the way the site is set up to let users move through the site and within 
individual pages. "Technical details" include features like slide shows, scroll bars, hyperlinks, quality or size of graphics, etc. 


Beginner: 1 

Developing: 2 

Mature: 3 

Professional: 4 

F: Navigation 

Navigation in the ePortfolio is confusing 
or awkward suggesting little consideration 
of user experience or limited technical 
skill. 

Navigation is not always consistent or 
intuitive. There are repetitions in navigation 
that do not enhance the user's experience or 
navigation choices that make the user's 
experience more difficult. 

Navigation is thoughtful and facilitates the 
user's experience, but isn't consistently 
sophisticated or effective. 

Navigation is thoughtful, supports the story 
being told, and is original/creative even if a 
template has been used. Navigation within the 
ePortfolio is sophisticated and improves the 
user's experience. 

G: Attention to 

Technical Details 

Major technical issues detract from the 
effectiveness of the ePortfolio. 

Minor technical issues occur, but these 
generally do not interfere with the 
effectiveness of the ePortfolio. Some 
features are inappropriate or do not 
contribute to the message or the 
construction of identity. 

No technical issues are detected with the 
ePortfolio. It is clear howto use the 
technical features of the ePortfolio. The 
features used are mostly appropriate and 
usually contribute to the message and 
construction of identity. 

No technical issues are detected, the features used 
are appropriate and consistently contribute to the 
message and identity. 

H: Ethical Literacy 

Choices in both texts and images are 
ethically problematic demonstrating little 
awareness of the public nature of 
ePortfolios or the necessity for respecting 
others in this public format. Citations are 
missing. 

Choices in either texts or images are ethically 
problematic. Questions arise about fair use, 
ownership, or appropriateness because of 
the lack of credits or or their inconsistency. 

Choices in either texts or images are mostly 
respectful of others and generally 
demonstrate an awareness of the public 
nature of ePortfolios. Credits and 
acknowledgement exist but are not 
consistent across the ePortfolio or a re not 
well-handled. 

Choices in texts and images consistently respect 
others and demonstrate awareness of the public 
nature of ePortfolios. No questions arise about fair 
use, ownership, or appropriateness because 
credits and acknowledgements are well-handled 
throughout the ePortfolio. 



AUBURN UNIVERSITY 

OFFICE OP UNIVERSITY WRITING 


Office of University Writing 

www.auburn.edu/writing 

auburnwrites@auburn.edu 

5/4/16 













Marshall, Bartlett, Duffy, and Powell 


Increasing Faculty Confidence 


204 


BPOMFOUO project Summative ePortfolio Rubric 

LEARN IT. LIVE IT. SHARE IT. 


Effective Communication 

Effective Communication is focused on the message of the overall ePortfolio rather than individual components. 


Beginner: 1 

Developing: 2 

Mature: 3 

Professional: 4 

1. Coherent Message for 
Intended Audience 

The textual and visual elements do not 
work in unison to construct a consistent 
identity and central message. The 
purpose of the ePortfolio as a whole is 
unclear or overtly clunky (1 want a job) or 
the reader has to do too much of the 
work to supply that message. 

Some elements of the ePortfolio as a 
whole obscure the message or create 
confusion about the purpose or the 
author's identity. There are several 
moments of public disclosure that do not 
serve to support the author's message or 
contribute positively to the construction 
of identity. 

The ePortfolio as a whole is almost always effective 
in both its message and the construction of identity. 
Most elements work well together so that the reader 
is drawn in and feels engaged in learning more about 
the author and his or her experiences, but there are 
some points where the pieces do not all work 
together to demonstrate sophistication in the 
message or construction of identity. 

There is a coherent and effective message 
being told by the ePortfolio as a whole and an 
identity is consistently and effectively 
constructed because all elements work 
together and demonstrate sophistication and 
originality/creativity. 
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Appendix C 
E-mail Invitations 


Dear [student name], 

As you know, Auburn University began an ePortfolio Project in 2012. We understand that you might have 
completed an ePortfolio during your time at Auburn. We are planning an Assessment Institute to have 
faculty members learn to evaluate ePortfolios and will publish the results of this assessment as part of a 
research study. We would like to use your ePortfolio in this Assessment Institute. If you agree to respond 
to the attached survey, which includes the opportunity to provide the URL of your ePortfolio, you will be 
entered into a drawing for a $50 Amazon gift card. All participants are also invited to come to the Office 
of University Writing (3436 RBD Library) to select a promotional item. 

Additional details of our study are provided below. Please read this consent information carefully and if 

you agree, follow the link to the survey provided at the end of this information. 


(NOTE: DO NOT AGREE TO PARTICIPATE UNLESS IRB APPROVAL INFORMATION WITH 
CURRENT DATES HAS BEEN APPLIED TO THIS DOCUMENT.) 

INFORMED CONSENT 
for a Research Study titled 
"ePortfolio Assessment and Faculty Development" 

You are invited to participate in a longitudinal research study to measure the improvement of student 
produced integrative, outward-facing, professional ePortfolios. The study will invite faculty members 
from the ePortfolio Project Faculty Cohort to attend an ePortfolio Assessment Institute where they will be 
trained to use the Project rubric and then read and score student ePortfolios from a variety of disciplines. 

The study is being conducted by Dr. Margaret J. Marshall, Director of University Writing in conjunction 
with Dr. Lesley Bartlett, Assistant Director of University Writing, and Dr. Megan Good, Director of 
Academic Assessment. You were selected as a possible participant because you are or were a student at 
Auburn who we believe produced an ePortfolio. 

What will be involved if you participate? If you decide to participate in this research study, you will 
answer a few questions about the experience of creating an ePortfolio and your use of that ePortfolio. You 
may also provide us with the URL of your ePortfolio(s) for use during the Assessment Institute. We will 
not be able to make changes to your ePortfolio (s). If you give us permission to use your ePortfolio, it may 
be used in future Assessment Institutes for comparative purposes. 

You will also have the opportunity to give us separate permissions to: 

• access your demographic information (major, GPA, gender, race, transfer status, first generation 
status, ACT/SAT, date(s) of changes in major) from the Office of Institutional Research (OIR) 

• quote from your ePortfolio or use screen shots of sections of your ePortfolio that could be 
captured without revealing your identity or revealing the URL to others 

• contact you at a later date for a short follow-up interview about your ePortfolio use 

Are there any risks or discomforts? The risks associated with letting us use your ePortfolio are breach of 
confidentiality and potential for psychological or social discomfort because faculty participating in the 
Assessment Institute will see your entire ePortfolio. 

Steps we will take to minimize these risks: 
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• We will ask faculty participants to sign an agreement that they will not share the URL to your 
ePortfolio with anyone and will not talk about the ePortfolios they assessed with anyone outside 
the Assessment Institute. 

• The Institute will be designed so that no faculty members from your major or minor are asked to 
read and score your ePortfolio. 

• If you give us permission to capture screen shots, we will ensure that your identity is not 
revealed and the screen shots do not include pictures of you or other identifying information. 

• Any demographic information you agree to let us access will be kept separate from your name 
and ePortfolio and will not be seen or used by faculty during the Assessment Institute. 

Are there any benefits to yourself or others? If you participate in this study, your ePortfolio will be used 
to gather assessment data that will serve to improve the ePortfolio Project and student learning at 
Auburn. 

Will you receive compensation for participating? As outlined above, you will be entered in a drawing 
with other participants to win a $50Amazon gift card. Only one gift card will be awarded each time we 
conduct an Assessment Institute. Your chances of winning depend on the number of participants but we 
estimate that number to be 1 in 200. All participants are also invited come to the Office of University 
Writing at 3436 RBD Library to select a promotional item of your choice. 

Are there any costs? There are no costs to you associated with this study other than the time you will 
spend responding to the survey. The survey should take you no more than 15 minutes. 

If you change your mind about participating, you can withdraw at any time during the study by closing 
your browser. If you decide to withdraw at a later date, you may contact Dr. Margaret Marshall at 
mjm0030@auburn.edu or at 334-844-7574. Your participation is completely voluntary. If at any time you 
choose to withdraw, all information and records of your participation will be deleted. Your decision 
about whether to participate or to stop participating will have no impact on your future relations with 
Auburn University, the ePortfolio Project, or the Office of University Writing. 

Your privacy will be protected. Any information obtained in connection with this study will remain 
confidential. Information obtained through your participation may be reported to faculty and 
administrative decision makers at Auburn and/or presented at conferences or published in scholarly 
journals but you will not be personally identified. 

If you have questions about this study, please contact Dr. Margaret J. Marshall at 
mmarshall@auburn.edu or 344-844-7474. You may print a copy of this document to keep. 

If you have questions about your rights as a research participant, you may contact the Auburn University 
Office of Human Subjects Research or the Institutional Review Board by phone (334)-844-5966 or e-mail at 
hsubjec@auburn.edu or IRBChair@auburn.edu . 
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HAVING READ THE INFORMATION PROVIDED, YOU MUST DECIDE WHETHER OR NOT YOU 
WISH TO PARTICIPATE IN THIS RESEARCH STUDY. 

IF YOU ARE WILLING TO PARTICIPATE IN THE SURVEY, ACCESSING IT WILL SERVE AS YOUR 
CONSENT TO PARTICIPATE. WHEN COMPLETING THE SURVEY, YOU WILL HAVE 
OPPORTUNITIES TO CONSENT TO THE LEVELS OF INVOLVEMENT OUTLINED ABOVE. 


Li nk to Survey 

_ 11-16-2015 

Primary Investigator Date 
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Appendix D 
Student Survey 


Student ePortfolio Al Research Survey 

Q1 I agree that I am at least 19 years old and want to participate in this research study. 
O No 
O Yes 

Q2 Thank you for taking the time to help the ePortfolio Project! If you complete this 
survey, you will be entered to win a $50 Amazon gift certificate. Your chances of 
winning are 1 in 200. Please stop by the Office of University Writing in the RBD Library 
to pick up a promotional item of your choice! 

Q 3 You may withdraw from this study at any tim e by contacting 
at or by phoning her at If you wish to withdraw 

during this survey, you may do so by closing your browser without hitting the submit 
button at the end of this survey and any answers that you have provided will be 
eliminated from the study. 

Q4 Please enter your first and last name below. 

First Name 
Last Name 


Q5 We are interested in how students use or plan to use ePortfolios when they 
transition to graduate school or professional careers. 
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Please select your response below. 



Strongly 
Disagree (1) 

Disagree (2) 

Neither Agree 
nor Disagree 
(3) 

Agree (4) 

Strongly Agree 
(5) ~ 

The process of 
creating an 
ePortfolio 
helped me 

O 

O 

O 

O 

O 

think about 
what 1 wanted 
to do after 
graduation. (1) 

1 used or plan 
to use my 
ePortfolio while 
looking for a 

O 

O 

O 

o 

O 

job or applying 
to graduate 
school. (2) 

Creating an 
ePortfolio 
helped me see 
connections 

o 

o 

o 

o 

o 

among my 
experiences. 

(3) 

Creating an 
ePortfolio 
helped me 

o 

o 

o 

o 

o 

explain my 
interests and 
skills. (4) 

1 have 

evidence my 
ePortfolio 
helped me 
secure a 

o 

o 

o 

o 

o 

position or 
admission to a 
graduate 
program. (5) 
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Display This Question: 

If We are interested in how students use or plan to use ePortfolios when they transition to graduate 
school or professional careers; I have evidence my ePortfolio helped me secure a position or admission 
to a graduate program. - Please select your response below. - Agree Is Selected 

Or We are interested in how students use or plan to use ePortfolios when they transition to graduate 
school or professional careers; I have evidence my ePortfolio helped me secure a position or admission 
to a graduate program. - Please select your response below. - Strongly Agree Is Selected 
Q6 What is the evidence your ePortfolio helped you secure a position or admission to a graduate 
program? 

Q7 I give the ePortfolio Project permission to use my ePortfolio(s) for the purposes described. Your 
identity will be visible to the faculty scoring your ePortfolio, but steps are in place to ensure that your 
participation is confidential. 

O No 
O Yes 


Display This Question: 

If I have read the Information Letter provided and give the ePortfolio Project permission to use my 
ePortfolio for the purposes described. Yes Is Selected 
Q8 How many ePortfolios do you currently have? 

O One 
O Two 
O Three 


Display This Question: 

If I have read the Information Letter provided and give the ePortfolio Project permission to use my 
ePortfolio for the purposes described. Yes Is Selected 
Q9 Please provide your ePortfolio link below. 

My ePortfolio URL: 

Approximate date of completion: 

Approximate date of last update: 


Display This Question: 

If I have read the Information Letter provided and give the ePortfolio Project permission to use my 
ePortfolio for the purposes described. Yes Is Selected 

And How many ePortfolios do you currently have? Two Is Selected 
Or How many ePortfolios do you currently have? Three Is Selected 
Q10 Please provide your second ePortfolio link below. 

My ePortfolio URL: 

Approximate date of completion: 

Approximate date of last update: 
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Display This Question: 

If I have read the Information Letter provided and give the ePortfolio Project permission to use my 
ePortfolio for the purposes described. Yes Is Selected 

And how many ePortfolios do you currently have? Three Is Selected 
Q11 Please provide your third ePortfolio link below. 

My ePortfolio URL: 

Approximate date of completion: 

Approximate date of last update: 


Q12 I give the ePortfolio Project permission to access the following demographic data from the Office of 
Institutional Research for the duration of this study only and in compliance with all FERPA regulations. 
The ePortfolio Project will keep your data stored separately from your name and ePortfolio URL. 
Anonymized data will be kept indefinitely._ 



Yes (1) 

No (2) 

Major (2) 

O 

o 

GPA (3) 

O 

o 

Gender (4) 

o 

o 

Race (5) 

o 

o 

Transfer Status (6) 

o 

o 

First Generation Status (7) 

o 

o 

ACT/SAT Score (8) 

o 

o 

Date of any changes in major (9) 

o 

o 
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Display This Question: 

If I give the ePortfolio Project permission to use my ePortfolio(s) for the purposes described. Yes Is 
Selected 

Q13 I give the ePortfolio Project permission to use anonymized parts of my ePortfolio for publications or 
conferences. The ePortfolio Project will only use quotes or anonymous screenshots in publications or 
presentations. Privacy settings on your ePortfolio will not be changed and steps will be taken to ensure 
that quotes or anonymous screenshots will not permit your ePortfolio to be found through common search 
engines if your privacy settings have prohibited public access. 

O No 
O Yes 


Display This Question: 

If I give the ePortfolio Project permission to use my ePortfolio(s) for the purposes described. Yes Is 
Selected 

Q14 The ePortfolio Project is interested in following up with students to find out about how students use 
ePortfolios after leaving Auburn. Please indicate if we may contact you later to ask follow up questions. 
O No 
O Yes 

Q15 I would like to be notified of any publications or reports using these survey results. 

O No 
Q Yes 
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Appendix E 

Pre/Post Faculty Confidence Survey 


One goal of the Assessment Institute is to foster a deeper understanding of the four student learning 
outcomes for the ePortfolio Project. We are interested in seeing how your understanding changes as a 


result of participating in the institute. 

Please respond 

to the following statements. 



Strongly 
Disagree (1) 

Disagree (2) 

Neither agree 
nor disagree 
(3) 

Agree (4) 

Strongly Agree 
(5) " 

1 feel confident 
teaching 

Critical 

Thinking 

O 

O 

O 

O 

O 

through 

Reflection in 
ePortfolios. (1) 

1 feel confident 
assessing 

Critical 

Thinking 

O 

O 

O 

o 

O 

through 

Reflection in 
ePortfolios. (2) 

1 feel confident 
teaching Visual 

o 

o 

o 

o 

o 

Literacy in 
ePortfolios. (3) 

1 feel confident 
assessing 

Visual Literacy 

o 

o 

o 

o 

o 

in ePortfolios. 

(4) 

1 feel confident 

teaching 

Technical 

o 

o 

o 

o 

o 

Competency in 
ePortfolios. (5) 

1 feel confident 

assessing 

Technical 

o 

o 

o 

o 

o 

Competency in 
ePortfolios. (6) 

1 feel confident 
teaching 

Effective 

o 

o 

o 

o 

o 

Communication 
in ePortfolios. 

(7) 

1 feel confident 

assessing 

Effective 

o 

o 

o 

o 

o 

Communication 
in ePortfolios. 

(8) 
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Appendix F 
AI Schedule 

ePortfolio Assessment Institute 
Auburn University 

DAY 1 Tuesday, May 10 th 

8:00 - 8:15 arrive, coffee, settle in 

8:15 - 8:45 introductions and details 

8:45 — 9:30 overview of ePortfolio Project with examples 

9:30 break 

9:45 — 11:30 understanding the summative rubric 

11:30-12:30 lunch 

12:30 - 3:20 training with the rubric 

3:20 break 

3:30 - 4:50 norming session 

5:00 collection of all materials and leave 

DAY 2 Wednesday, May 11 th 

8:00 - 8:15 arrive, coffee, questions from yesterday 

8:20 - 9:30 norming session 

9:30-2:00 scoring your packet 

11:30 lunch will be out; take your lunch and other breaks as you wish 

2:00-3:20 adjudication/discussion with your scoring partner 

3:20 break 

3:30-4:50 debrief on the experience of the Assessment Institute 
5:00 collection of all materials and leave 
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Appendix G 
Scoring sheet 


gPOMFOIIO project 

LEARN IT. LIVE IT. SHARE IT. 


SCORER: 

URL 


Critical Thinking Through 
Reflection 

Visual Literacy 

Technical Competency 

Effective 

Communication 

a 

b 

c 

d 

e 

f 

g 

h 

i 

EP 










Notes on ePortfolio 



Adjudication Notes 


SCORES AFTER ADJUDICATION 


Critical Thinking Through 
Reflection 

Visual Literacy 

Technical Competency 

Effective 

Communication 

a 

b 

c 

d 

e 

f 

g 

h 

i 

EP 
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